Skip to content

Conversation

kosabogi
Copy link
Contributor

This PR backports the following changes to the 8.x branch:
Adds 8.16 version to css matrix #115788

elasticsearchmachine and others added 30 commits October 17, 2024 14:54
The main users of this class use as input latitudes and longitudes read from doc values. These coordinates are always 
on bounds so there is no point to try to normalise them, more over when this piece of code is in the hot path for 
aggregations.
…lastic#115006) (elastic#115007)

This reverts commit 17ecb66 and
reapplies elastic#112881 once the
previous, non-backported transport version bump is dealt with.
As this induces ES-8275 and makes fleet time outs for some APIs.

Relates ES-9573
…ation (elastic#114946)

This commit makes the dot prefix deprecation match the existing changes to validation for the SLO
and SLA UIs.

Relates to elastic#112571
…ReaderOperator via BlockSourceReader. (elastic#114903)

Currently, in compute engine when loading source if source mode is synthetic, the synthetic source loader is already used. But the ignored_source field isn't always marked as a required source field, causing the source to potentially miss a lot of fields.

This change includes _ignored_source field as a required stored field and allowing keyword fields without doc values or stored fields to be used in case of synthetic source.

Relying on synthetic source to get the values (because a field doesn't have stored fields / doc values) is slow. In case of synthetic source we already keep ignored field/values in a special place, named ignored source. Long term in case of synthetic source we should only load ignored source in case a field has no doc values or stored field. Like is being explored in elastic#114886 Thereby avoiding synthesizing the complete _source in order to get only one field.
…lastic#115031)

* Bool query early termination should also consider must_not clauses

* Update docs/changelog/115031.yaml
Because of elastic#93575 it's not sufficient to mark repositories with
`readonly: true` while taking a backup. The only safe way to avoid
writes is to completely unregister them.
It's possible that the client config, particularly its credentials,
might change in the middle of a long-running operation such as a large
multipart upload. Prior to this commit we would hold onto the same
`AmazonS3` instance for the entire operation, but really there's no need
to do so, we can obtain a potentially-fresher instance for each API
call.
…apperTests testBlockLoaderFromColumnReaderWithSyntheticSource elastic#115074
…apperTests testBlockLoaderFromRowStrideReaderWithSyntheticSource elastic#115076
…tes (elastic#115075)

In JDK 23 `Thread.resume` has been removed this means that we cannot use
`IntermittentLongGCDisruption` that depends on it. 

We simulate the master node disruption with a `CyclicBarrier` that
blocks cluster state updates.

Closes: elastic#115045

The backport will close:
elastic#112634
Removes several more references to the now-unused `RestApiVersion#V_7`
constant, and decorates all remaining references with an `@UpdateForV9`
annotation so that they all have clear owners.
* docs: update synthetic source docs

* fix: also doc values false works

* Revert "fix: also doc values false works"

This reverts commit 0895a76.

* fix: update synthetic source documentation

* fix: all field types support it

* fix: no need to explicitly mention it

* fix: synthetic source sorting

* fix: may instead of might
When trying to use a gradle `skipTest` rule on tests named "values", we get a class-caste exception in Jackson.

This PR needs to rename this function for all versions of Elasticsearch that the `yamlRestCompatTestTransform` task will run on, so that later PRs that add skipTests will be able to pass. Since this test was added in 8.14, we must backport all the way back to there.

```
class com.fasterxml.jackson.databind.node.IntNode cannot be cast to class com.fasterxml.jackson.databind.node.ArrayNode (com.fasterxml.jackson.databind.node.IntNode and com.fasterxml.jackson.databind.node.ArrayNode are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader$InstrumentingVisitableURLClassLoader @50337c96)
```
…lastic#115082)

* Fix ML autoscaling (classic cloud) for models with zero allocations

* refactor a bit
This PR refactors the doc_values utils used in the TSDB codec to allow 
sharing between the current codec and the new codec.
…l exceptions (elastic#115017)

The model for calculating per-cluster `took` times from remote clusters in elastic#112595 was flawed. 
It attempted to use Java's System.nanoTime between the local and remote clusters,
which is not safe. This results in per-cluster took times that have arbitrary (invalid) values
including negative values which cause exceptions to be thrown by the `TimeValue` constructor.
(Note: the overall took time calculation was done correctly, so it was the remote per-cluster
took times that were flawed.)

In this PR, I've done a redesign to address this. A key decision of this re-design was whether
to always calculate took times only on the querying cluster (bypassing this whole problem) or
to continue to allow the remote clusters to calculate their own took times for the remote processing
and report that back to the querying cluster via the `ComputeResponse`.

I decided in favor of having remote clusters compute their own took times for the remote processing
and to additionally track "planning" time (encompassing field-caps and policy enrich remote calls), so
that total per-cluster took time is a combination of the two. In _search, remote cluster took times are
calculated entirely on the remote cluster, so network time is not included in the per-cluster took times.
This has been helpful in diagnosing issues on user environments because if you see an overall took time
that is significantly larger than the per cluster took times, that may indicate a network issue, which has
happened in diagnosing cross-cluster issues in _search.

I moved relative time tracking into `EsqlExecutionInfo`. 

The "planning time" marker is currently only used in cross-cluster searches, so it will conflict with
the INLINESTATS 2 phase model (where planning can be done twice). We will improve this design
to handle a 2 phase model in a later ticket, as part of the INLINESTATS work. I tested the 
current overall took time calculation model with local-only INLINESTATS queries and they
work correctly.

I also fixed another secondary bug in this PR. If the remote cluster is an older version that does
not return took time (and shard info) in the ComputeResponse, the per-cluster took time is then
calculated on the querying cluster as a fallback.

Finally, I fixed some minor inconsistencies about whether the `_shards` info is shown in the response.
The rule now is that `_shards` is always shown with 0 shards for SKIPPED clusters, with actual
counts for SUCCESSFUL clusters and for remotes running an older version that doesn't report
shard stats, the `_shards` field is left out of the XContent response.

Fixes elastic#115022
The elastic#113501 PR introduced a
change where `:index-name` is no longer considered a valid remote index
name. The valid remote index name has to have a non-empty remote cluster
name, e.g. `my-remote-cluster:index-name`.

This PR changes tests to avoid randomly generating empty remote cluster
names.

Resolves elastic#113659,
elastic#113660
elasticsearchmachine and others added 22 commits October 25, 2024 23:22
With Lucene 10, IndexWriter requires a parent document field in order to
use index sorting with document blocks. This lead to different IAE and file
leaks in this test which are fixed by adapting the corresponding location in
the test setup.
…urce/Date range} failing - Removed Old `Date range` test because it's not longer validating useful code (elastic#114057)

unmuting test and removing bwc test to get mixedClusterTest working
* Unmuting test issue elastic#115605

* fixing

---------

Co-authored-by: Elastic Machine <[email protected]>
@kosabogi kosabogi added >docs General docs changes Team:Docs Meta label for docs team labels Oct 29, 2024
@kosabogi kosabogi requested a review from leemthompo October 29, 2024 09:20
Copy link
Contributor

Documentation preview:

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-docs (Team:Docs)

@elasticsearchmachine elasticsearchmachine added external-contributor Pull request authored by a developer outside the Elasticsearch team v8.17.0 labels Oct 29, 2024
@kosabogi kosabogi closed this Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>docs General docs changes external-contributor Pull request authored by a developer outside the Elasticsearch team Team:Docs Meta label for docs team v8.17.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.